Letter to the Editor concerning “A systematic review of controlled trials on visual stress using intuitive overlays or colorimeter”

We read with interest the review written by Evans and Allen, and published in the Journal of Optometry, in July, 2016.1 Systematic reviews are considered the ‘gold-standard’ form of evidence for assessing the effectiveness of therapeutic interventions. A systematic review comprises a focussed question, a comprehensive search strategy to identify all potentially relevant studies, predefined selection criteria to minimise bias from ‘cherry-picking’ studies and an assessment of the risk of bias (RoB) of individual studies in a way that can be evaluated and reproduced. Because studies at high RoB often overestimate treatment effects,2 the aim is to either exclude studies at high RoB, or at least prioritise those studies at the lowest RoB. For this reason the RoB table is the key feature of any systematic review because it needs to inform the subsequent discussion. The authors state that they used the Critical Appraisal Skills Programme (CASP) checklist for assessing bias.3 However, the domains of bias outlined in Tables 2 and 3 do not correspond to the domains of bias of the CASP checklist.3 For example, in section-C of the CASP checklist for Randomised Controlled Trials (RCTs), three questions are posed. These are: (i) Can the results be applied in your context? (ii) Were all clinically important outcomes considered? (iii) Are the benefits worth the harms and costs? Users of the CASP tool are asked to provide Yes, No or Can’t Tell responses to each of these questions. In dealing with these three questions for the studies they reviewed, however, Evans and Allen chose to have only one column titled ‘‘Interpretation’’ in their tables, which mostly did not provide direct responses to the above section-C CASP questions. Thus the authors have developed their own, hybrid RoB rating scale which has unknown validity. Furthermore, the authors appear to have ‘cherry-picked’ the domains of bias used for each system;